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ABSTRACT 

This study, which took place in Maastricht 
(Netherlands), examined whether motivation affects the reliability of 
students' ratings about tutor behavior and whether different types of 
students have different expectations about tutor behavior. Four types 
of motivation were measured: intrinsic, extrinsic, and achievement 
motivation, and fear of failure. In addition, students' expectations 
about the importance of certain elements of tutor behavior were 
investigated. Student evaluation of instruction surveys were 
administered in 3 first-year courses in economics and business at the 
University of Limburg to 28 tutor-led small groups of 12 students 
each. The survey asked students to evaluate the performance of their 
small group's tutor. A total of 253 ^itudents completed the forms. 
Results showed that motivation was significantly, though weakly, 
related to students' ratings and students' expectations of tutor 
behavior. In addition, evaluations of tutor behavior were 
significantly correlated with intrinsic motivation, i.e., students' 
interest in course subject matter. Examination of the gap between 
students' perception of what tutors actually do and what tutors ought 
to do was found to be related to fear of failure. That is, the more 
students were afraid to fail, the less a tutor did what the student 
thought he/she ought to do and the more the student depended on the 
tutor's supportive role. (Contains 11 references.) (JB) 
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Abstract 



Student evaluations of instruction are a key instrument in research on tutor behavior in 
problem-based programs. The purposes of the present study are twofold. First, it 
attempts to examine whether motivatiai affects the reliability of students' ratings about 
tutor behavior. Second, it investigates whether different types of students have different 
expectations about tutor behavior. Four types of motivation were measured; intrinsic, 
extrinsic, and achievement motivation, and fear of failure. In addition, students' 
expectations about the importance of certain elements of tutor behavior were investigated. 
The results show that motivation is significantly, though weakly, related with students' 
ratings and students' expectations of tutor behavior. 
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Introduction 



Studies on tutoi effectiveness receive increasing attention in the literature on 
problem-based learning (e.g. Albanese & Mitchell, 1993). In general two kinds of 
studies may be identified: 1) studies v^'hich seek to identify essential charactraistics of 
tutor behavior that are required in problem-based tutorials, 2) studies that focus on degree 
of expertise in subject-matter required for effective tutoring. The first category contains 
empirical studies that are conconed with the question what kind of tutor skills are needed 
in the tutorial process to fulfil the tutor's tasks efficiently (e.g. Wilkerson, 1992; Moust, 
1993). These studies seek to identify skills that are important in guiding the work of 
discussion groups. For example, Moust (1993) found that an essential tutor skill is the 
tutor's ability to use vocational languap 'j congruent with students' level of competence. 
The second category consists of studies that examine whether tutors must be experts to 
realise a certain degree of directiveness in small group work (e.g. Davis, Nairn, Paine, 
Anderson & Oh, 1992; Moust, 1993). 

Student evaluations of instruction are a key instrument in research on tutoring, 
because of their proven reliability, validity and relatively low costs (Marsh, 1984; 
Gijselaers & Schmidt, 1991). Student ratings of tutor behavior may be regarded as 
descriptions of the instructional process going on in small groups. It is generally 
assumed that raters, students, are replicates (except for random error) in their measuring 
of the tutors' attributes or characteristics. That is, students within a group will give a 
similar rating on an item measuring an element of tutor behavior, except for some random 
error. The amount of random error determines the reliability of ratings. Research has 
shown that random error -or also called within-group variability- is normally modest or 
small, depending on the number of students and quality of items, resulting in sufficiently 
high degrees of reliability. For example, Gijselaers and Schmidt (1991) found that 
interrater-reliabilities within small groups (when based upon 7 ratings) is normally above 
.80. In genejal, this is regarded as adequate to get reliable tutor behavior measures. 

Whether differences in ratings between students within groups are to be interpreted 
as random or as systematic, has been an issue of considerable debate in the literature on 
student ratings (see for an extensive review, Feldman, 1977). The issue is whether 
dissimilarities in ratings not only reflect random error, but also systematic error. If the 
latter is the case, then some of the patterned variability in ratings represents so-called true 
variance. Differences among students may then result from legitimate or genuine sources 
of influences on their ratings. However, studies on the relationship between certain 
attributes of students have in general shown only weak relations between ratings and 
these attributes. For example, Feldman (1977) mentions that only students' motivation 
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and expected grade appear to be consistent related to student ratings of instructions. 
Marsh (1984) found that the effect of Prior Subject Interest on student ratings of 
instruction was greater than that of any of 15 other background variables considered. 
The Prior Subject Interest variable was most highly correlated with students' ratings of 
the course's learning value (rs about .4). Ratings with dimensions of teaching behavior 
were lower (rs between .3 and -.12). According to Marsh (1984), higher student interest 
in the subject matter apparently created a more favorable learning environment and 
facilitated effective teaching. Comparable results were found in a study by Jones 
(1981). He showed that students' ratings of teaching are in part related to what they 
consider as good teaching. Students' criteria about what constitutes good teacMng may 
differ depending on their basic expectations of the course. Highly motivated students 
tended to give more favorable ratings for the same course than poorly motivated students. 
In general, it may be concluded from reviews of the student ratings literatwe (Feldman, 
1977, Marsh, 1984) that effects of background variables tend to be small, except for 
motivation variables and students expectations. 

The purpose of the present analysis is twofold. First, it attempts to examine 
whether motivation variables affect the consistency of ratings about tutor behavior. It 
tries to clarify the association between types of raters (for example highly motivated Vs 
poorly motivated) and ratings of tutor behavior. The second purpose of this study is to 
investigate whether different types of students have different expectations about how a 
good tutor should behave and if he/she actually performs this behavior in educational 
practice. Consequently, knowledge about the influence of motivation variables on tutor 
behavior may enhance a better understanding and interpretation of students' evaluative 
ratings of tiie behavior of tutors. 

In the present study four types of motivation were measured; intrinsic motivation, 
extrinsic motivation, achievement motivation and fear of failure. In addition, students' 
expectations about the importance of certain elemraits of tutor behavior were investigated- 
The pi esent study describes findings of a study conducted at the problem-based school of 
economics ^fid business at the University of limburg, the Netherlands. 

Method 

Subjects. During the 1993-94 academic year at the University of Limburg, 
Maastiicht, the Netherlands, student evaluation of insti^ction surveys were administered 
in 3 first year courses of the problem-based school of economics and business. Student 
evaluation instruments were collected in 28 small groups, consisting of 12 students each. 



An average of 75% of the students (=253 students) enrolled in small groups completed 
the survey forms. 

Description of the curriculum. The four-year economics and business curriculum 
consists of a four seven weeks course paiods. Each course follows the same problem- 
based format. Students met with their tutor in small-group tutorials, twice a week for 
two hours, to analyse and discuss problems that are related to the general multidisciplinar 
course theme. At the beginning of each seven-week course period students were 
randomly assigned to tutorial groups. 

Instrument. At the end of each course a questionnaire was administered before the 
examination. Courses usually contained 25 groups, representing 12 students for each 
group. In every course 10 or 9 small groups were asked to fill out the questionnaire. 
The questionnaire contained two sections. The first section on tutor behavior consisted 
of three-point likert scale items (categories ranging from 1 = "tutor insufficiently 
showing the particular behavior" to 3 = "tutor showed behavior in a sufficient way") 
about various aspects of tutor's behavior. The section on tutor behavior consisted of 19 
items measuring specific elements of tutor behavior and one overall evaluation item. 
Previous validation research showed that three dimensions were underlying the specific 
tutor behavior items: 1) knowledge about subject-matter'of the course, 2) procedural 
teaching skills for small group work, 3) tutor's motivation and interest in students' 
learning experiences (Dolmans et al, 1993). The internal consistency of the "Subject- 
matter Knowledge" scale was .74 (8 items), for the "Tutor's Teaching Skills" scale (6 
items) .62, for the Tutor's Motivation scale (5 items) .65. The inter-rater reliability 
(intraclass-correlation) when based upon 7 ratings within a group was above .80 for the 
individual scales. In addition, students were asked to rate for every specific tutor 
behavior item the necessity of that behavior: "l=behavior is unimportant for problem- 
based tutorials", "2=behavior is appropriate, but not very important for problem-based 
tutorials", and "3=behavior is important for problem-based tutorials". Exhibit 1 contains 
a sample of the section I items on tutor behavior. 

The second section of the questionnaire contained motivation items taken from an 
inventory (Nuy, 1991) that aims to measure study approaches in problem-based learning 
(I.S.A.P.L.). This inventory was developed within the research tradition on learning 
styles of college students (e.g. Schmeck, 1988). The motivation categories were: 1) 
Intrinsic motivation (Interest in study content; challenged by questions and problems), 2) 
Achievement Motivation (Need to excel; high standards of achievement), 3) Fear of 
failure (Avoidance of stress and uncertainty), and 4) Extrinsic motivation (Instrumental 
function of studying). Each category contained 8 Likert-type items. The internal 
consistencies of the individual motivation scales were above .75. 
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Exhibit 1: Sample of items on tutor behavior in section 1 of the evaluation survey. 



Tutor behavior. 



As a description of this tutor: 

1 = tutor insufficiently showing 

the particular b^vior 

2 = Neutral 

3 = tutor showed behavior in a 

sufficient way 



Knowledge about subjea- matter 
of the course 

• Hie tutor appeared to be 
aware of the principles of 
problem-based learning. 

• The tutor ^coinaged a 
thorough review of the 
problems aft^ studying 

• The tutor displayed a fair 
understanding of the course's 
contents. 



Prccedural teaching skills for 

small group work 

The tutor stimulated 
evaluation of tutonal group 
meetings. 



Tutor's motivation and interest 
in students' learning experiences 
The tutor invited students to 
express their own opinions 
and ideas. 



The necessity of this behavior 
was: 

1 = behavior is unimportant for 

problem-based tutorials 

2 = behavior is appropriate, but 

not very important for 
problem-based tutorials 

3 = behavior is important for 
problem-based tutorials 



1 2 3 



1 2 3 



1 2 3 



1 2. 3 



Analysis. Data were analysed at the individual student's level. Student ratings 
were summarized by scores on the three tutor behavior scales and four motivation scales. 
Also a measure called the "Gap factor" was computed that takes into account the 
difference between the rating on a tutor behavior item and the necessity of that tutor 
behavior: 

G = ( Behavior - Importance 

The size and direction of the gap factor serves as an indication of students' 
perceptions of the extent whether a tutor behavior was congruent with students' 
expectations. In general three situations may occur. 

Situation I: l^\ing0ehavior) = ^^B(importancr) => G « 0 
Situation II: ^t\ng0ehavior) ^ ^^mdmportance) =» G ^ 0 
Situation ID: Rating(2«'/iav»or; ^ ^^Z(importance) =» G < 0 
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A negative gap factor indicates that tutor's behavior was less in the direction than 
students' regarded as important for problem-based learning. A positive gap factor 
indicates the opposite: a tutor showed behavior more than needed. A zero gap factor 
suggests that tutor's behavior was congruent with student's needs. 

Results 



Table 1: Correlations between motivation and tutor behavior variables. 







Intrinsic 
Motivation 


Extrinsic 
Motivation 


Achievement 
Motivation 


Fear for 
Failure 


1. 


intrinsic Motivation 


1.00 








2. 


Extrinsic Motivation 


-.46** 


1.00 






3. 


Achievement Motivation 


.41** 


-.11 


1.00 




4. 


Fear of &ilxire 


-.04 


.36** 


.04 


1.00 


5. 


Tutors' Subject-matter Knowledge 


.15* 


-.07 


-.01 


-.06 


6. 


Tutor's Teaching Skills 


.15* 


-.04 


.08 


-.12 


7. 


Tutor's Motivation 


.11 


-.03 


.03 


-.06 


8. 


Gap-factor Subject-matter Knowledge 


-.05 


.01 


-.01 


-.18** 


9. 


Gap-factor Tutor's Teaching Skills 


-.03 


-.04 


.08 


-.13* 


10. 


Gap-factor Tutor's Motivation 


-.06 


-.03 


.03 


-.21** 



Note * - Signif. LE .05 N = 253 

**- Signif. LE .01 

As mentioned in the introduction, the purposes of the present study are twofold. 
First, it attempts to examine whether motivation variables affect the consistency of 
ratings about tutor behavior. The second purpose of this study is to investigate whether 
different types of students have different expectations about how a good tutor should 
behave and if he/she actually paforms this behavior in educational practice. 

The first issue is whether ratings are linked to differences in student types within 
classes or discussion groups. As Feldman (1977) pointed out: "together with the fact 
that students are not trained as either observers or raters, it might well be expected that 
ratings done in typical classroom settings would be dependent to some extent on the 
characteristics and experiences of the student observers". In this particular case, the 
major question is whether differences in types of motivation influence students' ratings 
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of tutor behavior. This question can be approached by simply correlating students' 
evaluation of tutoring with measures of motivation. 

Table 1 contains the simple correlations between the four motivation variables, gap- 
factors and students' ratings of tutor behavior. The findings demonstrate that student's 
intrinsic motivation is weakly related to student's ratings of tutor's subject knowledge 
and teaching skills. Students who are more intrinsically motivated tend to give slightly 
higher ratings (.2 on a 3-point scale) for tutors' subject-matter knowledge and tutor's 
teaching skills. Extrinsic motivation and achievement motivation are not related to 
ratings of tutor behavior. Correlations among the motivation variables show that 
intrinsic motivation is positively related to achievement motivation. Fear of failure is 
associated with extrinsic motivation. Intrinsic motivation seems to be a variable that 
influences some aspects of effective tutoring. Higher student into-est in subject-matter of 
a course apparently creates a more favorable learning environment and hence facilitates 
effective tutoring. A related interpretation is that higher student interest may positively 
influence student's observation of tutor bdiavior. 

As mentioned in the method section, conelational analyses were conducted at the 
individual level. However, in the present case interpretation of the correlational results 
must be made with caution. Simple correlations masks much of the systematic variance 
that may exist between various tulmal groups. Although random assignment tot tutorial 
groups took place, it may be possible that some groups contained more highly (or 
poorly) motivated students than might be expected by chance alone. Feldman (1977) 
pointed out that, 'the possibility exists that certain kinds of "fit" (both perceived and 
actual) between teachers and different students in their groups are related to ratings'. 
This is the possibility of interaction effects between types of motivation, or composition 
of the discussion group as pertaining to motivation, and various tutors in different 
discussion groups. Consequently, students' motivation may be differentially affected by 
specific discussion groups or tutors. 

Analyses of variance provide an approach to address the issue whether, next to 
main effects of motivation, a main effect of discussion group and interaction effects 
exists. ANOVA makes it possible to determine whether the obtained correlations are 
confounded by effectr of tutorial groups. In the present study, student rating data were 
analyzed at the class level through two-way analyses of variance. These analyses were 
designed to address the question whether the conelations between motivation and 
student's ratings of tutor behavior were confounded by effects of various discussion 
groups. Table 2 and 3 contain examples of the results of the two-way analyses of 
variance. In this particular main effects of motivation and discussion groups were 



independent variables. Students' rating of tutor subject-matter knowledge and tutoring 
skills were dependent variables. 

Td)le 2: Tutors' Subject-matter knowledge 



Source of Variarion 


Sum of 
Squares 


DF 


Mean 
Square 


F 


Sig 
of F 


Main Effects 


10.675 


30 


.356 


5.162 


.000 


Intrinsic motivation 




O 
J 






.VJiO 


Discussion group 


9.675 


27 


.358 


5.198 


.000 


2-Way Interactions 


4.408 


68 


.065 


.940 


.606 


Intr. Mot * Disc, group 


4.408 


68 


.065 


.940 


.606 


Explained 


15.083 


98 


.154 


2.233 


.000 


Residual 


10.616 


154 


.069 






Total 


25.699 


252 


.102 







253 cases were processed. 



Table 3: Tutors ' Teaching Skills 





Sum of 




Mean 




Sig 


Source of Variation 


Squares 


DF 


Square ' 


F 


ofF 


Main Effects 


12.725 


30 


.424 


5.720 


.000 


Intrinsic motivation 


.688 


3 


.229 


3.094 


.029 


Discussion group 


11.017 


27 


.408 


5.502 


.000 


2-Way Interactions 


4.902 


68 


.072 


.972 


.544 


Intr. Mot * Disc, group 


4.902 


68 


.072 


.972 


.544 


Explained 


17.628 


98 


.180 


2.425 


.000 


Residual 


11.421 


154 


.074 






Total 


29.049 


252 


.115 






253 cases were processed. 













The results in table 2 and 3 show that significant main effects exist for intrinsic 
motivation and discussion group. No interaction effects were found. The degree of 
association between motivation and students' ratings of tutor behavior may be calculated 
as r|2 (which is equal to the proportion of variance explained). For table 2 r|2 = 
.715/25.699 = .028, which is comparable with a simple correlation of .16. For table 3 
r|2 = .688/29.0499 = .024, which is comparable with a simple correlation of .15. Both 
estimates of r|2 show that the simple correlations in table 1 are not inflated by effects of 
discussion groups. Correlations and r|2 are the same. Similar results were found for 
other relations between the motivation variables and students' ratings of tutor behavior. 
In conclusion, the main effects motivation and discussion group are independent. 
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Hence, calculations of correlations at the individual level provide adequate estimates of 
the relation between motivation and students' evaluations of tutor behavior. 

The second purpose of this study was to examine whether relations exist between 
students' expectations about education and ratings of tutor behavior. For each item of 
the evaluation survey the gap between the ideal and the actual tutor behavior was 
measured. The size of the gap factor serves as an indication of students' perceptions of 
the extent by which the actiial behavior falls short of the ideal (Jones, 1981). The results 
in table 1 show that students witii a higher degree of fear of failure tend to give slightiy 
negative scores on the gap factor. That is, the more students indicate they have fear of 
failure, these students perceive that actual (or perceived) tutor behavior is less 
corresponding with tutor behavior being regarded as necessary. This means that 
students with high fear of failure view tutor behavior, as operationalized in our 
questionnaire, as less sufficient than required. Interestingly, other motivational 
background characteristics are not related to the score on the gap factor. 

Discussion and Conclusion 

The present study attempts to examine whether motivational background is related to 
students' ratings of tutor behavior. The focus of tiiis shidy is twofold: first it tries to 
clarify whether students are interchangeable as raters (is witiiin-class variability due to 
random error alone?), second it seeks for relations between types cf students and 
elements of tutor behavior that are perceived as essential for problem-based learning. 
With respect to the first purpose of this study it was found that evaluations of tutor 
behavior were significantiy correlated, though modest to w^k, with intrinsic motivation 
(students' interest in course subject matter). Other motivation variables were not related 
to ratings of tutor behavior. Whether tiiis correlation is interpreted as "rating bias" or 
"actiial different behavior of the rated object" depends in large part on tiie acclaimed 
objectiveneft of student ratings. It might be the case that highly intiinsic students indeed 
see different things in tutor behavior than students who are less interested in a course. Or 
anotiier interpretation may be that highly intrinsic motivated students create a more 
favorable learning environment, which in turn positively influences tutor behavior. 

The second purpose of this study was to examine whether students have different 
expectations about tutor behavior, given their motivational background. So-called gap- 
factors for ttitor behavior (the gap between "is" and "ought") were calculated and related 
to motivation. The relative size of the gap-factor may be interpreted as an indication of 
the difference between what students perceive as important and how tutoring ac'cually 



took place. The results showed that fear of failure was consistently, though modest, 
related with the gap-factors. A relative high degree of fear of failure was associated with 
negative gap-factors. That is, the more students are afraid to fail, the less a tutor does 
what he should do in the opinion of these students. Students with a high degree of fear 
of failure consider the supporting role of tiie tutor as \ery important 

It was shown that different groups of students tend to perceive learning environment 
differently with respect to how tutors should behave. An additional outcome of this 
study is that students' evaluations of actual tutor behavior aie to a very low degree 
affected by motivational backgroimd. As such, this latter outcome corresponds with 
previous research on the reliability of student ratings. The present study shows that 
patterned variability in student ratings rqjresents to some extent true variance and not 
systematic error. Expectancies of students, as a result of students' motivation, may be 
regarded as valuable information to interpret ratings of tutor behavior. Clearly, the 
results from the present study ask for further in-depth studies on the role of motivation in 
tho functioning of problem-based small groups. 
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